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The NASA Constellation Program has worked for the past five years to develop a re- 
placement for the current Space Transportation System. Of the elements that form the 
Constellation Program, only two require databases that define aerodynamic environments 
and their respective uncertainty: the Ares launch vehicles and the Orion crew and launch 
abort vehicles. Teams were established within the Ares and Orion projects to provide repre- 
sentative aerodynamic models including both baseline values and quantified uncertainties. 

A technical team was also formed within the Constellation Program to facilitate integra- 
tion among the project elements. This paper is a summary of the collective experience 
of the three teams working with the quantification and use of uncertainty in aerodynamic 
environments: the Ares and Orion project teams as well as the Constellation integration 
team. Not all of the lessons learned discussed in this paper could be applied during the 
course of the program, but they are included in the hope of benefiting future projects. 

Introduction 

Development of aerodynamic environments for the Constellation flight vehicles has not occurred without 
significant technical challenges. One of those challenges has been the development of high-confidence aerody- 
namic databases including quantified uncertainty. This paper discusses lessons learned from addressing those 
technical challenges during the development stage of the flight vehicles. The paper contains the collective 
experience of teams working aerodynamic uncertainty quantification from both the project and program 
levels. While some of the lessons learned have a broader application, the focus here is on the quantification 
and use of aerodynamic uncertainties. Lessons learned are grouped into five overarching themes: 

Lesson I: Plan and Re-Plan Often 

Lesson II: Develop an Operational Strategy 

Lesson III: Integration is Critical 

Lesson IV: Manage Uncertainty Engineering Processes 

Lesson V: Consequences of Epistemic Uncertainty 

The presentation of lessons learned is arranged from high program/project level to low team level and 
in order of importance as perceived by the authors. Lessons for the program level are directed to the 
development of requirements and system architectures. Project level lessons are geared to development and 
evaluation of a system with respect to defined requirements. Team level lessons focus on the quantitative 

* Research Aerospace Engineer. Configuration Aerodynamics Branch (CAB). Senior Member AIAA. eric.l.walker@nasa.gov 
t Senior Research Aerospace Engineer. CAB. Associate Fellow AIAA. michael.j.hemsch@nasa.gov 
■^Research Aerospace Engineer. CAB. Senior Member AIAA. jeremy.t.pinier@nasa.gov 

§ Research Aerospace Engineer. Aerothermodynamics Branch. Senior Member AIAA. karen.l.bibb@nasa.gov 
^Research Aerospace Engineer. CAB. Member AIAA. david.t.chan@nasa.gov 
II Research Aerospace Engineer. CAB. Member AIAA. jeremy.l.hanke@nasa.gov 

1 of 14 


American Institute of Aeronautics and Astronautics 



description of the system. Some of the lessons presented are painfully obvious in hindsight, while others 
are still counter-intuitive. At the onset of the Constellation Program, the teams involved did not have a 
full appreciation for what would be required to implement uncertainty quantification in the aerodynamic 
databases. This paper is the result of a series of conversations that occurred from 2006 to 2011 within 
the Ares and Orion Aerodynamic Teams as well as the Aerodynamic Team within the Constellation Flight 
Performance System Integration Group. An important perspective to keep in mind while reading is that 
the aerodynamic databases, including the uncertainty statements, are in a constant state of flux during the 
lifespan of the projects. This perspective was best stated by a colleague: 

Aerodynamics , the first thing a project needs an estimate of, and the last thing to get fully defined. 

— Philip Robinson, Orion Aerodynamic Database Coordinator, NASA Johnson Space Center 

Lesson I. Plan and Re-Plan Often 

In preparing for battle, I have always found that plans are useless but planning is indispensable. 

— Dwight D. Eisenhower 

Hindsight has a way of making things that should have been done obvious to those who needed to do 
them, but only after it is too late to make a difference. Often the most pressing issues that the uncertainty 
teams encountered were either lack of time in the schedule to do an uncertainty analysis, lack of information 
or data necessary to determine uncertainty, or, as was often the case, both. This was especially true in the 
early stages of the project when the schedules did not allow for sufficient time, if any, to perform the analysis 
tasks necessary to build a representative uncertainty model. In addition, early planning did not account 
for additional acquisition or generation of data for the purpose of quantifying sources of uncertainty. Even 
after the teams began to dedicate resources to uncertainty development in planning phases, there was still 
difficulty with discovery. When additional sources of uncertainty were discovered, it often became difficult 
to incorporate these within cost and on schedule, as resource budgets were determined one to two years 
before most of the data generation or acquisition activities took place. With a large number of issues to be 
addressed, reassigning enough contingency resources to get what was really needed for the uncertainty models 
became difficult. At times, this led to the necessity of becoming very creative in how various uncertainty 
terms were estimated. Planning for uncertainty development activities needs to take place at all levels. 

A. Program Level: Uncertainty Drives Design — Pay Now or Pay Later 

One of the main purposes of an uncertainty estimate is to determine if the vehicle design is in danger of not 
meeting requirements. Requirements can be established for a variety of reasons, but primarily to prevent 
loss of crew and/or loss of mission. The Constellation Program did not directly place any requirements on 
the aerodynamics of vehicles within the new space transportation system architecture. Requirements were 
specified for vehicle performance to meet the demands of the design reference missions. The vehicle perfor- 
mance requirements were assessed by customers of the aerodynamic database which included the guidance, 
navigation, and control teams and the structures and loads communities. These customers performed their 
assessments using the uncertainty statements to either disperse the aerodynamic environment along with 
other aspects of the vehicle and flight trajectory, such as wind profiles and mass properties, or to develop 
worst-case design conditions. 

Because it is usually cheaper to make design changes early in a vehicle program before flight hardware has 
been built and certified, it is advantageous to have appropriately defined uncertainty models very early in the 
design and analysis cycles or even as early as the conceptual phase. Further discussion of the conceptual phase 
is contained in Lesson III. A. The brevity of the conceptual phase will create difficulties for new projects or 
programs to introduce additional capability to the conceptual design tools and be able to take full advantage 
of them. Enhancement of conceptual design tools will most likely require base funding from the organization 
or the availability of off-the-self tools with these capabilities. Until these tools can be enhanced to provide 
realistic or somewhat reasonable uncertainty estimates, it is recommended that programs should plan to 
allocate resources specifically for early definition of uncertainty in the first few design and analysis cycles to 
maximize benefit. If these uncertainties are defined in a realistic manner, as a general rule the uncertainties 
should not drastically increase as the design matures unless the design itself significantly changes. 

There is no guarantee that uncertainties will decrease as the design matures unless there is a managed 
margin burn-down plan as mentioned in Lesson III.A. This is predicated on the class of uncertainty that is 
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encountered. As discussed in Lesson V, there are two major classes of uncertainty: irreducible (aleatory) 
and reducible (epistemic). Irrespective of margin, uncertainties will decrease if and only if they are of the 
reducible class. Not planning resource allocation for early development of uncertainties will force the sole use 
of margins in design early on and limit insight into the design trade space. Understanding the uncertainty 
model and how the lack of knowledge or the inherent variability of environments affects the vehicle is critical 
to understanding the vehicle design requirements, performance characteristics and, ultimately, risk. 

B. Project Level: Strike a Balance Between Baselining and Uncertainty Development 

It is better to solve the right problem approximately than to solve the wrong problem exactly. 

— John Tukey 

Fundamentally, uncertainty quantification allows for approximate definition of the aerodynamic environ- 
ments of a given flight vehicle. Planning for the development of a quantified uncertainty at the project 
level cannot be emphasized enough. Uncertainty quantification costs money and in most cases must not 
be ignored. The big question here is: “How much will it cost?” This can be quite difficult to determine 
when the physics of the aerodynamics are complicated and there are more than a few important independent 
variables. Depending on that complexity, the uncertainty quantification can be a substantial portion of the 
cost of defining the environments. 

Ground testing and computational simulation of the aerodynamic environments are only approximate 
tools. Each of these tools has multiple levels of fidelity, but neither are exact. This defines a trade where the 
project must decide how to balance resources between establishing an approximate baseline and quantifying 
the uncertainty of that baseline. When deciding how to allocate resources to this balance, tremendous 
benefit can be gained in knowing what the vehicle performance metrics are most sensitive to. For new types 
of vehicles the knowledge of sensitivities can be limited or non-existent early on but, when similar vehicles 
exist or have existed, there should be some level of experience concerning the predominate sensitivities. 
As the design matures intimate interaction with the customers of the aerodynamic database, primarily the 
guidance, navigation, and control and the structures and loads communities, will yield additional insight 
into the vehicle sensitivities. 

C. Team Level: Balance Uncertainty Term Definition 

A man loses his keys and is looking for them under a lamppost. His friend comes over and asks 
“ Where did you last see them?” 

Man replies, “ I think I dropped them by my car.” 

Friend asks, u Then why are you looking for them over here?” 

Man answers, “ Because the light is so much better.” 

— Anonymous 

Quantifying one uncertainty source to high-confidence is not helpful if another uncertainty source of 
comparable impact is poorly defined and/or estimated. This problem occurred in the Ares and Orion 
projects many times as the complexity of the databases increased. The information available for building a 
new portion of the database and estimating its uncertainty was often sparse and relatively low in confidence. 
Inadequately defined databases and uncertainty modeling often resulted in the need to use engineering 
judgment to obtain uncertainty estimates. 

A prime example of the need to balance uncertainty term definition can be taken from the development 
of the Ares I Stage Separation Aerodynamic Database. Using data generated from a wind tunnel simulation 
of unpowered stage separation of the Ares I launch vehicle , 1 several months were spent developing a high- 
fidelity unpowered aerodynamic database. During the review process (see Lesson IV. B), concern was raised 
regarding the level of uncertainty of the powered effects. A low- fidelity adjustment for the powered effects 
was made to the high-fidelity unpowered database, and the resulting uncertainty for the low-fidelity powered 
effects adjustment dwarfed the high-fidelity unpowered uncertainty model. In this case, better planning 
could have allowed resources to be allocated to address the powered effects of the stage separation event. 
Sadly, this was not an isolated incident. It is easy to fall into the trap of spending the most time measuring 
and evaluating the uncertainty where most of the data exists instead of concentrating on better defining 
sources contributing most of the uncertainty. 
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Lesson II. Develop an Operational Strategy 

All the business of war , and indeed all the business of life , is to endeavor to find out what you 
don’t know by what you do; that’s what I called ‘guessing what was at the other side of the hill. ’ 

— Arthur Wellesley, 1st Duke of Wellington 

One of the end goals of any strategy to quantify aerodynamic uncertainty is to minimize or eliminate 
the need for guessing; though guessing can still be a tempting and potentially necessary option at times. 
This section discusses a framework of strategies for a more formalized development of quantified uncertainty. 
This framework is necessary to consistently treat and interpret those uncertainties. There are a number of 
examples of the uncertainty models used in the development of the aerodynamic databases for Ares 2,3,4,5,6, 7,8 

and Orion. 9,10 ’ 1 ! 

A. Program Level: Create a Permanent Team 

Aerodynamic engineers and analysts need to be well-trained in uncertainty quantification methods and 
gain experience with real projects where data are messy, often inadequate and resources are limited. It is 
acceptable to outsource tool development, but the team must integrate the tools into real projects and find 
the gaps that need further work. While far better to develop this team before the start of a project, Ares and 
Orion did not have this luxury. Ideally, if there are multiple programs and/or projects, the team should have 
members at various stages in the design cycle to maximize exposure and to minimize unnecessary duplication 
of effort. 

There are several other aspects of the team that are worth discussing. The team should not be isolated 
from its discipline or customers. As discussed in Lesson III.C, it is imperative that the team understand 
all aspects of how data are derived. Isolation from the discipline makes this understanding more difficult 
and can lead to misunderstanding. Conversely, keeping the members of the uncertainty quantification team 
integrated with the rest of the aerodynamics team accomplishes several things: it keeps the members’ skills 
sharp within the discipline, makes them aware of advances in the field, creates an environment for innovative 
solutions to difficult problems, serves to train others in uncertainty quantification methods, and aids in 
planning activities to help ensure necessary data are acquired. 

B. Project Level: Individuals Responsible for Data and Uncertainty 

Uncertainty work should not be exclusively performed by the uncertainty quantification team. At minimum, 
those providing data to the process should learn how to quantify the uncertainty of their respective compo- 
nents of data. This is important because insight into processes and their respective assumptions by those 
actually generating data makes these individuals or groups the best candidates to determine the uncertainty 
of their data, with guidance from the uncertainty quantification team. This responsibility ties into Lesson 
V that the uncertainty expression cannot be divorced from its baseline value. 

C. Team Level: Use a Multi-level Uncertainty Modeling Strategy 

It should be recognized early on in the project that all sources of uncertainty cannot be quantified in the same 
way. The uncertainty teams within the Constellation elements of Ares and Orion classify three primary types 
of uncertainty based on the amount of information about the terms and how readily they can be quantified. 
The three categories, described below, are as follows: (1) measurement-estimated uncertainty; (2) recognized 
uncertainty and (3) unknown uncertainty. 

The first category, measurement-estimated uncertainty, refers to sources of uncertainty that lend them- 
selves to a statistical measurement of variation. The teams used methods for measuring variation that were 
previously applied by Hemsch et a /. 12,13 to standard artifact testing in the wind tunnel facilities at NASA 
Langley Research Center for the purpose of quantifying force balance repeatability and reproducibility. These 
methods are based on statistical process control 14,15,16 and range analyses 17, 18 to measure variation. The 
teams applied these variation measurement techniques to repeat data acquired during wind tunnel tests 
as well as multiple computational fluid dynamics simulations of the same conditions and geometry. More 
detailed examples and analysis can be found in the references of the Ares and Orion teams given above. 

The second category is recognized uncertainty, which refers to sources of uncertainty that are recognized 
but either few or no data are available for the measurement of variation. An example of this type of 
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uncertainty is discussed by Bibb et al. 9 with respect to a vaguely- defined geometry for an asymmetric heat 
shield for the Orion Crew Module. The baseline uncertainty model for the Crew Module was developed 
using a symmetric heat shield geometry and the project needed to be sure that an asymmetric heat shield of 
the several proposed approximate geometries would not violate vehicle flight performance requirements. An 
engineering analysis was performed to determine a reasonable extent to which vehicle forces and moments 
would be impacted. 

The last category is that of unknown uncertainty of which is difficult to give a specific example, but 
important to acknowledge its existence. The teams used a term or multiple terms depending on the particular 
situation to embed a specified amount of margin into the uncertainty models based on engineering judgement. 
Further discussion of the introduction of margin into the uncertainty models is given in Lesson II. I and Lesson 

III. A. 

D. Team Level: Aerodynamic Uncertainty Modeling = Plausible Physics / Statistics 

That a conclusion reached in one environment (say the laboratory) will apply in a different en- 
vironment (say the full-scale process) is based not on statistical reasoning but on what Deming 
called ‘a leap of faith.’ Good statistics and subject matter knowledge can narrow the chasm but 
not eliminate it. 

— George Box 

The ultimate goal of an aerodynamic uncertainty model is to determine the extent to which physics can 
be misrepresented by the respective aerodynamic database. Only the measurement-estimated type of the 
three types of uncertainty previously discussed is determined with statistical methods. The remaining two 
types are estimated with the use of engineering analysis and judgment. For the statistical methods that 
are used in this work, simple robust calculations that are easily explained and defended are used wherever 
possible. More advanced methods are used only when necessary. 

Another important aspect of the aerodynamic uncertainty model is that even though descriptive statistical 
terms are used, they are implemented in the model in a predictive sense. In general, prediction of the 
flight vehicle performance is an inference not a statement of fact, thus the resulting uncertainty model for 
aerodynamics is primarily epistemic in nature (See Lesson V for further discussion). Care must be taken when 
interpreting the results of these inferences as this goes far beyond the basic use of probabilistic distributions. 

E. Team Level: Use Operational Definitions for Every Process — Be Explicit 

There is nothing as deceptive as an obvious fact. 

— Sir Arthur Conan Doyle 

The old adage, “Say what you mean and mean what you say,” is absolutely critical for transparency and 
clarity; however, in practice, this is often taken for granted. For this reason and others expressed in Lesson 
III, every nominal value, every uncertainty statement and every process that is carried out must be explicitly 
defined in such a way that someone else can obtain the same result using the defined process. This is why it 
is best to do substantiation documentation during the development phase even though schedules are always 
tight. When processes are explicitly defined, transparency, trust and the ability to more readily reproduce 
results are enhanced. Such definitions also help to find misunderstandings and holes in analysis. This is 
deceptively simple and straightforward, but individuals often inadvertently assume too much about what 
was intended by others, as stated above. 

F. Team Level: Quantify All Sources of Uncertainty 

All sources of uncertainty should be quantified, even if it is believed that their effects are small. The 
final uncertainty bounds should be quantified along with an explanation of how it was done, especially if 
engineering judgment was used. In general, reliance on engineering judgment (see Lesson II. I) should be 
avoided unless the uncertainty is tiny. This will obviously only address the first two categories of uncertainty 
discussed previously in Lesson II. C. 
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G. Team Level: Traceability for the Generation of the Final Reported Models 

This record needs to be as detailed as possible. It should include each time the data is manipulated. This 
is a painstaking process but is worthwhile. Good traceability for the aerodynamic model generation process 
will accomplish two things. First, it will reflect the number of times the data have been manipulated, each 
time allowing error and uncertainty to creep into the process. Secondly, the record will make most of the 
uncertainty bookkeeping obvious, clarifying if terms are necessary, have been duplicated, or neglected. 

H. Team Level: Realistically Estimate Correlation and Incorporate It, When Necessary 

You can see a lot by just looking. 

— Yogi Berra 

Very often, because of lack of time or data, assumptions are made that correlation of terms is either 
perfect or non-existent when it is likely that it is neither. There is a tendency to use whichever of these 
situations is more advantageous to the problem at hand. Correlation or covariance is a very broad topic most 
often associated with error in physical measurements, and many methods exist to estimate it . 19 Hemsch and 
Walker 5 discuss the correlation of uncertainty terms that arise in construction of increments or corrections 
derived from computational fluid dynamic simulations. To provide a realistic uncertainty model, evaluation of 
the covariance or correlation of terms is important to determine whether or not they are clearly independent. 
Failure to incorporate correlation can result in over- or under-prediction of uncertainty, either of which can 
cause problems when the aerodynamic databases are implemented in performance simulations. The Orion 
project has several examples that include estimated and appropriately implemented correlation terms in the 
aerodynamic database uncertainty models for the Crew Module 9 and the Launch Abort Vehicle . 11 

I. Team Level: Reserve Engineering Judgment for Including Margin In Uncertainty Models 

It is a capital mistake to theorize before one has data. Insensibly one begins to twist facts to suit 
theories, instead of theories to suit facts. 

— Sir Arthur Conan Doyle 

Of the three types of uncertainty discussed in Lesson II. C, the first two should be evidence-based, to the 
fullest extent possible. Once evidence has been exhausted, then and only then should the teams begin to 
rely on engineering judgement. Discretion must still be used with the evidence-based models to ensure that 
they are realistic in nature. Even evidence-based models can be in error due to misinterpretation of data. 

J. Team Level: Develop Strategies for Uncertainties Where No Data Exist 

To this point, the discussion has focused on what to do in areas where data exist. In developing aerodynamic 
databases, it is also important to express to the customer what is expected to occur in areas where data 
do not exist. There can be many reasons why there are no data available for a particular region of the 
database. For example, early in the process there will often be very sparse coverage. Another possibility 
is that a particular region of the database could not be addressed due to limited resources. Even so, the 
responsibility falls to the aerodynamics team to address these regions using engineering judgment. If there 
is no specification of data given, the customer will be forced to invent it for themselves. This can lead to 
a very untenable situation because there are often multiple customer groups, and the chances are slim that 
each group will invent identical data. 

While extrapolating data is difficult enough, uncertainties must also be defined in extrapolated regions. 
Therefore, basic strategies for addressing those uncertainties are important. There are no guarantees for 
situations like these and, if these regions become important, they should be added to the program and/or 
project risk mitigation plan whichever is most appropriate. There are several things to consider here. Is the 
model that is being used to represent the aerodynamic environments purely statistical or is it mechanistic? 
Statistical models are based on a calibrated parametric or non-parametric mathematical representation of 
the data. In contrast, mechanistic models are based on mathematical representation of underlying physical 
mechanisms. The uncertainties of extrapolated statistical models, especially those of a non-parametric or 
locally-fit variety, tend to be large. For mechanistic models, uncertainty growth is not necessarily so drastic. 
As long as the extrapolation does not cross a flow physics boundary, it is not unreasonable to continue use 
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of the same or similar levels of uncertainty. Once that boundary is crossed, however, extrapolation of the 
uncertainties is untenable. Thus, to safely extrapolate uncertainties, it may only be necessary to verify or 
justify that no physics changes are expected in the extrapolation region. 

Lesson III. Integration is Critical 

We should not be afraid of finding out something. 

— George Box 

Interfacing with customers is one of the most critical aspects of the design and analysis cycle-the more 
efficient the communication, the more productive the design cycle. Some may inquire why a discussion about 
integration is in a paper on uncertainty. The reason is simply this - failure to communicate the intent of an 
environment database and its uncertainty results in implementation uncertainty. 

It is also important to foster a working environment where teams can have a straight-forward technical 
discussion without fear of retribution. If this type of working environment does not exist, it can result in 
margin being hidden in the data products and issues not being brought to the attention of managers or other 
teams in a timely manner. Teams will have a tendency to be defensive if they feel that their products may 
be used inappropriately. 

A. Program Level: Manage Uncertainty, Margins and Safety Factors at a High Level 

One must learn by doing the thing ; for though you think you know it , you have no certainty until 

you try. 

— Sophocles 

Changing from the current margin plus safety factor engineering culture to one of designing under uncer- 
tainty is a real revolution. Revolutions carry huge risks during transition. As the transition takes place, care 
must be taken to not stumble into unknowns that were previously masked by margins and safety factors. 
Fear associated with changes in culture can create resistance, sometimes overt, sometimes passive. It may 
require several design cycles before the culture has changed enough to realize significant benefits. Complete 
culture change may not be realized until the previous generation of engineers have left the organization. 

Another major question to be asked here is, “What should a ‘design under uncertainty’ culture look 
like?” In the authors’ opinion, uncertainty may not completely replace the current margin plus safety factor 
culture. Instead, it is anticipated that the components be used synergistically. For example, uncertainty 
models should be subject to some level of margin to be held at a high level within a program or project. 
This is primarily to allow for unknowns in immature designs and complex systems. A well-managed burn 
down of margin in the uncertainty model should allow for a gradual reduction in the levels of uncertainty as 
knowledge is gained from one design cycle to the next. There should be a conscious effort to avoid drastic 
increases in uncertainty due to early under-prediction of uncertainty combined with a late discovery of an 
unexpected physical phenomena. Overall, the change in culture should make risk management an easier 
proposition by reducing the guess work. 

This leads to another point of interest regarding conceptual design tools. Even though tools used in 
conceptual design are typically reduced-order, the numbers generated must have a reasonable uncertainty 
expression that is based on experience and physics. This underscores another need for the integration of 
disciplines. These statements of uncertainty should be embedded into the tools so that engineers using the 
tools are confronted with them. Investing in uncertainty expressions for conceptual design tools should help 
lower the overall cost of uncertainty development in the preliminary design phase. One further reason for 
investment in conceptual design uncertainty statements is the ability to perform risk-informed trades earlier 
in the project, thus allowing an earlier assessment of vehicle viability with respect to the reference missions. 

B. Project Level: Never EVER Throw Anything Over the Wall 

Not everything that can be counted counts and not everything that counts can be counted. 

— Albert Einstein 

Teams generating uncertainty models for data must understand how the resulting models will be used. 
They must also ensure that they are being used and results interpreted as intended. In other words, teams 
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must not generate data products and then simply “throw them over the wall” to the customers without 
working to ensure that the models are implemented correctly. Varying levels of misunderstanding and 
misuse of models between the aerodynamics teams and their customers occurred within both the Ares and 
Orion projects. There was often inadequate integration and/or documentation. Misuse and misinterpretation 
of models results in unquantified (and, perhaps, unrecognized) uncertainty and corresponds directly to an 
undocumented increase to program risk. 

The issue of misunderstanding and misuse of aerodynamic environments and their corresponding uncer- 
tainty models seemed to have been more of a problem for the Ares Project than the Orion Project. Database 
delivery method is believed to be a contributing factor for the issue. The Ares Aerodynamics Team chose 
to provide their database products to customers by way of a table of values in a spreadsheet with caveats 
on appropriate use and documentation on how to combine values from multiple tables for simulation of 
various flight conditions and vehicle configurations. With this choice of delivery, each customer of the Ares 
Aerodynamic Database implemented the calculations for the aerodynamic environments for themselves. 

In contrast, the Orion Aerodynamic Database implementation involved a series of tables in a standardized 
format used in conjunction with a compiled library of functions to assemble the tables and a defined data 
structure for passing the results to the customer application, through an application programming interface 
(API). The API was delivered with documentation on the contents of tables, the definition of the interface, 
and a list of caveats for appropriate use. This method only required the customers of the Orion Aerodynamic 
Database to implement the interface to the API but not to perform the detailed calculations to assemble 
the aerodynamic environment themselves. 

C. Team Level: Apply Uncertainty in the Appropriate Context 

Rule 1. Original data should be presented in a way that will preserve the evidence in the original 
data for all the predictions assumed to be useful. 

Rule 2. Any summary of a distribution of numbers in terms of symmetric functions should not 
give an objective degree of belief in any one of the inferences or predictions to be made therefrom 
that would cause human action significantly different from what this action would be if the original 
distributions had been taken as evidence. 

— Walter Shewhart, Rules for Presenting Data for Prediction 

Uncertainty is context dependent. When uncertainty is quantified, it is applicable only to a particular 
region. Teams quantifying uncertainty must understand, to a fault, how the information given to them was 
generated. Quantifying uncertainty is part of a system involving processes which are interconnected and 
must work together. Most of these processes are embodied in the people generating the data. 

D. Team Level: Create a Credibility /Maturity Scale 

Once the aerodynamic database has been released, the customers, namely the guidance, navigation and 
control and structures and materials communities, do not readily have access to information regarding the 
credibility and maturity of each part of the database. The aerodynamic database will almost always contain 
data and models that are at widely different levels of fidelity and maturity. As discussed in Lesson I.A, 
requirements were not directly levied on the vehicle aerodynamics but on performance. This requirements 
structure necessitated that the aerodynamics teams work closely with the guidance, navigation and controls 
teams. As issues arose with performance simulation, requests would be made to the aerodynamics teams to 
refine results where possible. This iterative process is necessary because the understanding of fidelity and 
maturity is embodied in the people that generated the data and models. 

Though not in any of the current databases, a solution has been proposed to provide additional informa- 
tion to the customers of the aerodynamic database. The teams have proposed the use of a credibility /maturity 
scale, using the NASA Standard for Models and Simulations 20 or Oberkampf and Roy 21 as guidance, that 
would be included with the database. This information could be used by the customers to determine when 
it is best to ask for a refinement of an area of the database to potentially reduce uncertainties with data 
and/or analysis, or when to make the design more robust. 
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Lesson IV. Manage Uncertainty Engineering Processes 

A pessimist sees the difficulty in every opportunity ; an optimist sees the opportunity in every 
difficulty. 

— Winston Churchill 

During a program or project, it can often seem overwhelming to manage the amount of information that 
is present at any given time. One of the most important lessons learned during the Constellation Program 
was that the uncertainty engineering processes need to be managed. This aspect of management goes over 
and above the margin discussion in Lesson III. A. The overarching processes involved in the development, 
use and interpretation of uncertainty need to be managed at all levels. Not managing these processes will 
not necessarily result in damage to the program/project but will severely limit the impact of additional 
insight provided by these processes, including quality control. Another point to stress is that the uncertainty 
engineering process is an overlay to the existing engineering process that provides an additional set of 
information and tools to aid all levels of management in a more efficient use of resources, as well as the 
aforementioned quality control process. 

To reiterate, the results of managing uncertainty in the engineering process are threefold: 

1. Quantitative estimation of uncertainties. 

2. Quality control of the data acquisition and database development processes. 

3. Additional depth of insight into existing processes. 

It is believed that the uncertainty engineering processes will prove to be a valuable addition to the current 
engineering process. 

A. Program Level: Create an Uncertainty Management Team 

Although the Constellation Program never officially enacted this type of team, the need for such a team 
had become evident and discussions were in progress near the end of the program. From these preliminary 
discussions, this team should consist of several members that have sufficient insight into the technical dis- 
ciplines and their interfaces such that they can manage use and interpretation of uncertainty statements 
incorporated into the engineering processes. One of the major issues that arose during the program was 
the interpretation of performance simulations with respect to the probabilistic verification requirements. 
Technically astute uncertainty management should understand the various approaches to simulation of per- 
formance, with uncertainty expressions, relative to requirements and be able to provide guidance with the 
verification requirements to explicitly define the framework for interpretation. 

B. Project Level: Manage Review Processes 

A review process should be structured to ensure the technical quality of the databases. Both the Ares and 
Orion projects had relatively elaborate review processes. Even with the processes that were in place, there 
was still some concern that error could creep into the aerodynamic database development process. High- 
level programmatic review cannot guarantee that error will not occur at the lowest level. Many errors in the 
actual computations can be missed during these high-level reviews, by being masked in the way the material 
is presented to the review panel. Our recommendation here is to determine the most sensitive or critical 
areas of the database and ensure that detailed technical reviews are performed and the resultant database 
model can be independently generated. 

Figure 1 shows a high-level flow diagram of the Constellation element review processes for aerodynamic 
databases. This figure is reflective of what the processes evolved to near the end of the program. 

For Ares, figure 1(a), the programmatic components of the review process are shown in red and the 
independent components are shown in purple. The database working group of the Ares Aerodynamic Team 
would submit a portion of the database for review to a branch-level independent technical review. The 
primary task of the branch-level review was to determine if the material under review was produced in a 
manner consistent with standard practices. The database portion was then referred to a directorate- level 
independent technical quality review, which was tasked with determining the suitability of the resultant 
database portion for inclusion in the overall database and reviewing and recommending any caveats on the 
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intended use of the material. These results were sent to the Ares Aeropanel for final review and release to 
the database customers for evaluation. Post evaluation, the newly updated database and the results of the 
evaluation were submitted to the Ares Ascent Flight System Integration Group for acceptance as the new 
aerodynamic baseline for the project. 

The flow of the review process was similar for Orion, figure 1(b), with a notable exception. Before 
release to the database customers for evaluation, the Orion Aerodynamics Team held a formal database 
release candidate review and invited the Aerosciences Technical Discipline Team (TDT) from the NASA 
Engineering Safety Center (NESC). The portion of NESC-TDT that attended the release candidate review 
consisted of both internal and external experts in aerodynamics related to atmospheric flight of spacecraft. 
There were no practical differences in how new databases and their evaluations were treated between Orion 
and Ares. The only other noteworthy aspect of the review process was that the branch- level and directorate- 
level review teams had some reviewers in common to both the Ares and Orion, which helped maintain 
consistency. 

C. Team Level: Configuration Manage Analyses and Data Files 

Mistakes are painful when they happen, but years later a collection of mistakes is what is called 
experience. 

— Denis Waitley (American motivational Speaker and Author of self-help books, b.1933) 

Mistakes are bound to happen in any development process, and it is generally accepted that programmatic 
resources will not allow for every aerodynamic database that is constructed to be independently verified. The 
viewpoint of the team should be that any and every analysis is subject to repetition. With this viewpoint 
in mind, analyses should be scripted or otherwise designed to be executed multiple times. These tools and 
their respective data files should be kept under configuration management to help ensure overall quality. 

D. Team Level: Develop Standard Processes and Tools 

Part of team management should be to work with the project and program uncertainty managers to determine 
appropriate standard practices and processes. To streamline analysis and help combat implementation errors, 
common analyses should be developed into tools or at least a library of routines that can be used throughout 
the team as databases are being developed. 

Lesson V. Consequences of Epistemic Uncertainty 

As alluded to earlier in the paper, there are two classes of uncertainty, aleatory and epistemic, that 
depend on how the uncertainty arises in the system. For the purpose of this paper, definitions for aleatory 
and epistemic uncertainty are adapted from Oberkampf and Roy 21 and included below: 

Aleatory uncertainty is inherent variation associated with a parameter, physical system, or environment 
and is also referred to as variability, stochastic uncertainty, and irreducible uncertainty. It is a characteristic 
of the system being analyzed. Some examples include variability in geometric parameters from one flight 
article to another due to manufacturing processes and variability in weather conditions. 

Epistemic uncertainty arises from imperfect knowledge or ignorance and is also referred to as subjective 
uncertainty, reducible uncertainty, or model-form uncertainty. It is a characteristic of the state of knowledge 
of the analysis team. Examples of epistemic uncertainty include insufficient experimental data to precisely 
characterize a probability distribution and poor or limited understanding of physics phenomena or physics 
coupling. 

The remainder of this lesson discusses how the consequences of epistemic uncertainty were addressed and 
provides some recommendations. 

A. Program Level: Classify Uncertainty 

From the perspective of aerodynamic uncertainty, a primary objective should be to quantify the uncertainty 
so as to comprehensively inform the resultant risk of the vehicle or architecture meeting the specified mission 
requirements. Fundamentally, the benefits of classification result in the ability to better risk-inform system 
level trades and resource allocation. The design of the system architecture must be robust to aleatory 
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or irreducible uncertainty but not necessarily to epistemic uncertainty. Classifying the uncertainty and 
understanding where it exists and where the architecture is sensitive to it will aid in the allocation of 
resources to appropriately reduce the uncertainty, modify requirements, or redesign a component of the 
system. This classification of uncertainty is used in the nuclear power reactor safety community, 22 storage 
of nuclear waste 23 and the risk assessment community. 24,25,26,27,28,29,30 

B. Project Level: Uncertainty Over-prediction 7^ Conservative Prediction of Risk 

Discovering the unexpected is more important than confirming the known. 

— George Box 

It is generally assumed that using larger uncertainties results in a more conservative design. This assump- 
tion is not necessarily accurate. It must be taken into consideration that uncertainties are being propagated 
through a system that may not have a linear response to inputs. The typical approach to performance 
simulation in the Constellation Program elements of Ares and Orion was to treat all uncertainty components 
as aleatory regardless of classification, primarily because there was no classification for the aerodynamic 
uncertainties. This treatment of all uncertainties as aleatory is consistent with Pate-Cornell’s 26 Level-4 
probabilistic risk analysis. Pate- Cornell 26 also discusses a Level-5 probabilistic risk analysis with multiple 
risk curves that segregates the aleatory and epistemic uncertainties in the analysis. The primary reason 
for implementing a Level- 5 analysis is tendency of the Level-4 analysis to mask risk if significant epistemic 
uncertainty is present in the system. It is also important to note that just increasing an incorrectly classified 
uncertainty will not necessarily change the risk levels substantially as a result of the central tendency in the 
aleatory only analysis. Thus, over-prediction of uncertainty incorrectly used can still mask risk. 

Our recommendation to counteract the possibility of hiding risk is three- fold. First, continue to treat all 
uncertainty as aleatory in performance analysis simulations. Second, rank order the inputs to the simulation 
to identify the most sensitive. Third, repeat the performance analysis with the most sensitive inputs treated 
according to their classification. This recommendation is derived from the launch risk acceptability process 
for national test ranges. 31 

C. Team Level: Point-based Uncertainty Models are Limited 

Use of point-based estimates of uncertainty makes it difficult to build spatial or parametric correlation into 
the dispersions. There is a need to move to field-based dispersion models to better replicate physical behavior. 

Bias shifting of the uncertainty from the nominal or baseline value to the nominal plus or minus the 
uncertainty may not be sufficient to adequately stress the vehicle simulation. Simply shifting the nominal 
value from the database may not reflect what the flight vehicle will experience. If the vehicle is more sensitive 
to gradients, then the uncertainty model should allow that aspect of the vehicle response to be stressed in 
the simulation. 

Dispersion models provided with aerodynamic databases should be explicit in their functionality. The 
problem here is that the default dispersion method of bias up/down offsets to the nominal data can lead 
to unrealistic results (over-conservatism) and/or not stress the aerodynamic model in enough ways. More 
sophisticated, physically constrained dispersion models should be derived and implemented in the flight 
simulations to better represent the vehicle behavior in flight. Pinier 8 discusses this issue at length and 
provides examples. 


Final Remarks 

In an attempt to highlight the major lessons learned by the various teams involved in the aerodynamic 
uncertainty development, some of the subtleties and nuances of the uncertainty development had to be left 
out of this paper. Most of the lessons that were included here were the result of a hard-fought battle to 
understand what was needed and what would help. Since not all of the lessons and recommendations were 
actually implemented during the Constellation Program, it is fully anticipated that adjustments will need to 
be made when these recommendations are actually used. Future papers will address these details in depth. 
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(a) Ares aerodynamic database review process. (b) Orion aerodynamic database review process. 


